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Attached for your action is our final report, Transportation Security Administration's 
Screening of Passengers by Observation Techniques. We incorporated the formal comments 
from the Transportation Security Administration in the final report. 

The report contains six recommendations aimed at improving the Screening of Passengers 
by Observation Techniques program. Your office concurred with all recommendations. 
Based on information provided in your response to the draft report, we consider the 
recommendations resolved and open. Once your office has fully implemented the 
recommendations, please submit a formal closeout letter to us within 30 days so that we 
may close the recommendations. The memorandum should be accompanied by evidence 
of completion of agreed-upon corrective actions and of the disposition of any monetary 
amounts. 

Consistent with our responsibility under the Inspector General Act, we are providing copies 
of our report to appropriate congressional committees with oversight and appropriation 
responsibility over the Department of Homeland Security. We will post a redacted version 
of the report on our website. 

Please call me with any questions, or your staff may contact John E. McCoy II, Deputy 
Assistant Inspector General for Audits, at (202) 254-4100. 
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Executive Summary 

We audited the Transportation Security Administration's (TSA) Screening of Passengers 
by Observation Techniques program. The program's intent is to screen passengers by 
observing their behavior in order to detect potential high-risk travelers. This program 
uses Behavior Detection Officers to detect passenger behaviors that may be indicative 
of stress, fear, or deception. Congressman Bennie Thompson requested an audit of 
TSA's Screening of Passengers by Observation Techniques program to determine its 
effectiveness, efficiency, and economy as a security screening protocol at airports. The 
audit objective was to determine whether TSA's Screening of Passengers by Observation 
Techniques program is structured to ensure that passengers at U.S. airports are 
screened in an objective and cost-effective manner to identify potential terrorists. 

Since the Screening of Passengers by Observation Techniques program began in fiscal 
year 2007, data provided by TSA indicate that the program has expended an estimated 
$878 million and has more than 2,800 full-time equivalent positions, as of September 
30, 2012. However, TSA has not implemented a strategic plan to ensure the program's 
success. For example, TSA did not (1) assess the effectiveness of the Screening of 
Passengers by Observation Techniques program, (2) have a comprehensive training 
program, (3) ensure outreach to its partners, or (4) have a financial plan. As a result, 
TSA cannot ensure that passengers at United States airports are screened objectively, 
show that the program is cost-effective, or reasonably justify the program's expansion. 
In fiscal year 2012, TSA's Behavior Detection and Analysis Division developed a draft 
strategic plan that includes a statement of mission, goals, and objectives. However, the 
plan had not been approved and implemented at the time of our review. 

We made six recommendations to improve the effectiveness of the Screening of 
Passengers by Observation Techniques program. TSA concurred with all 
recommendations. 
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Background 

According to TSA, it conducts passenger and baggage screening for approximately 
1.8 million passengers daily at more than 450 TSA-regulated U.S. airports. TSA officials 
stated that TSA developed and implemented behavior detection as a layer of security 
(see Figure 1) in response to the Implementing the Recommendations of the 911 
Commission Act of 2007. The program emphasizes objective behavior observation and 
analysis techniques to identify potentially high-risk individuals who are engaged in some 
form of deception and fear discovery. 1 According to TSA, the Passenger Assessment 
Screening System was based on the behavior assessment concept used by Israeli 
security forces. 



Figure 1. Layers of U.S. Aviation Security 




Source: TSA 

In 2002, Boston's Logan Airport and the Massachusetts State Police/Port Authority 
piloted the Passenger Assessment Screening System. In 2003, TSA developed the 
Screening of Passengers by Observation Techniques (SPOT) program with assistance 
from the Massachusetts State Police. TSA began operational testing at Logan Airport in 
Boston, Massachusetts, and later expanded its testing at two other New England 
airports. According to the U.S. Government Accountability Office (GAO), in fiscal year 



1 The Screening of Passengers by Observation Techniques Standard Operating Procedure establishes 
procedures and standards for the uniform screening of individuals. 
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(FY) 2007, TSA implemented SPOT, and was authorized 644 Behavior Detection Officers 
(BDOs) for deployment to 42 airports. According to TSA, as of FY 2012, more than 3,000 
BDOs were authorized for deployment to 176 U.S. airports. TSA's Behavior Detection 
and Analysis Division is responsible for developing strategic plans and program guidance 
for the SPOT program. 



BDOs, working in pairs, primarily 
conduct SPOT at airport screening 
checkpoints by having brief verbal 
exchanges with passengers waiting in 
line, while observing passengers' 
behaviors. A BDO identifies passengers 
for additional screening based on an 
evaluation system of identified 
behaviors, which may require a referral 
to law enforcement. A law 
enforcement officer (LEO) may assess 
the situation by interacting or engaging 
the passenger to determine if law 
enforcement intervention is necessary 
(see Appendix C for more details on the 
SPOT referral process). Passengers 
whose observed behaviors are not 
resolved during the referral process may 
not be permitted to board an aircraft. 




A TSA BDO speaking with an airline traveler. BDOs 
often converse with passengers to detect clues 
regarding suspicious or deceptive behavior from 
travelers before they go through metal detectors. 



Source: Aram Boghosian, The Boston Globe 



TSA BDOs are required to document all relevant information regarding each passenger 
referral in a referral report and the Performance Measurement Information System 
(PMIS). However, the SPOT Standard Operating Procedure prohibits the collection of 
personally identifiable information in the referral report or PMIS. Reports on the results 
of SPOT operations, such as the SPOT Situation Report Dashboard - All Airports (see 
Appendix D for an example of this report) are generated from the data entered into 
PMIS. According to TSA, SPOT referrals made from October 2011 through September 
2012 resulted in 199 arrests (shown in Figure 2). The reasons for arrests included 
outstanding warrants, suspected drugs, and illegal aliens. 
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Figure 2. SPOT Referrals and Arrests, October 2011 through September 2012 



Source: TSA data 2 

* Passenger throughput estimated at more than 450 airports, of which 176 use SPOT. 

The SPOT program was concentrated at large TSA regulated airports. 3 According to TSA, 
the insufficient coverage at smaller airports left a gap in TSA's additional layers of 
security. 4 In FY 2012, the program was expanded to smaller airports to increase the 
number of passengers potentially screened by BDOs as they enter the air transportation 
system. 

In May 2010, GAO reported deficiencies in the SPOT program and made 11 
recommendations; see Appendix E for further details about these recommendations. 5 
In addition, the SPOT program has come under scrutiny from Congress and the media. 
The Honorable Bennie G. Thompson, Ranking Member, Committee on Homeland 
Security, U.S. House of Representatives, requested that the Office of Inspector General 
(OIG) conduct an audit of TSA's SPOT program to determine its effectiveness, efficiency, 
and economy as a security screening protocol at airports. 

Results of Audit 

Since the SPOT program began in FY 2007, data provided by TSA indicate that the 
program has expended an estimated $878 million and has more than 2,800 full time 



2 This information is based on TSA's SPOT Situation Report Dashboard -All Airports, which may be 
incomplete and inaccurate due to errors identified in the PMIS. 

3 TSA classifies its regulated U.S. airports into one of five categories— X, I, II, III, and IV. Category X airports 
generally have the largest number of passengers boarding planes and category IV airports have the least. 

4 Behavior Analysis Capability Risk Based Allocation Methodology Phase I: Draft Report, July 2012. ■ 



5 U.S. Government Accountability Office, GAO-10-763, Efforts to Validate TSA's Passenger Screening 
Behavior Detection Program Underway, but Opportunities Exist To Strengthen Validation and Address 
Operational Challenges, May 2010. 



657,000,000 
Passenger Throughput FY 2012' 
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growth. However, TSA has not implemented a strategic plan to ensure the program's 
success. 

An agency's strategic plan is a valuable tool for communicating a common vision to 
managers, employees, partners, Congress, and the public. Above all, it should be used 
to align resources and guide decision making to accomplish priorities and improve 
outcomes. TSA did not (1) assess the effectiveness of the SPOT program, (2) have a 
comprehensive training program, (3) ensure outreach to its partners, or (4) have a 
financial plan. As a result, TSA cannot ensure that passengers at U.S. airports are 
screened objectively, show that the program is cost-effective, or reasonably justify the 
program's expansion. In FY 2012, TSA's Behavior Detection and Analysis Division 
developed a draft strategic plan that includes a statement of mission, goals, and 
objectives. However, the plan had not been approved and implemented at the time of 
our review. 



Figure 3. SPOT Program Growth (Estimated expenditures in millions) 
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Performance Management 



TSA cannot accurately assess the effectiveness or evaluate the progress of the 
SPOT program. The program does not have a finalized strategic plan that 
identifies the mission, goals, and objectives needed to develop a system of 
performance measures. Even though program operations and referral results 
are recorded in the PMIS, the referral data captured are incomplete and 
inaccurate. Unreliable PMIS data cannot be used to measure performance or 
produce an accurate account of SPOT activity. As a result, TSA cannot 
demonstrate that BDOs are screening passengers in a uniform manner to 
identify potentially high-risk individuals. Office of Management and Budget 
(OMB) Circular A-ll emphasizes that an agency's strategic plan should define its 
missions, goals, and the means by which it will measure its progress. Specifically, 
strategic plans should— 

• Specify the outcomes that the agency wants to accomplish; 

• Ensure that each strategic goal is supported by performance goals or 
measures; and 

• Ensure that performance goals set priorities and targets, and track 
progress in achieving the goals. 

Performance Measures 

TSA had not developed performance measures for the SPOT program. The 
program's Standard Operating Procedure indicates that its purpose is to identify 
high-risk individuals who may pose a threat to transportation security. The 
program collects data from referral reports that provide measurable outputs of 
specific activities. However, these outputs do not provide a measure of program 
effectiveness, because TSA has not established why these outputs support 
desired outcomes. For example, TSA documents the identification of prohibited 
items, undeclared currency, and illegal aliens, but the SPOT program has not 
defined how these outputs support achieving the SPOT program goals. As a 
result, TSA cannot assess the SPOT program's success without relative outcome- 
oriented performance measures. In August 2012, TSA provided its draft Behavior 
Detection and Analysis Division Performance Metrics Plan. This plan is intended 
to identify current gaps in performance metrics collection, proposed metrics 
solutions, and resource requirements for the next 3 years. The plan had not 
been approved and implemented at the time of our review. 
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Data Collection 

Although program operations and outputs from referrals are recorded in the 
PMIS, the referral data captured are not always complete or accurate. BDO 
managers are required to review and approve all SPOT referral reports entered 
into PMIS, but the internal controls over data entry were ineffective. For 
example, PMIS will automatically bypass manager review of a referral report 
after 72 hours. There were insufficient data entry quality assurance measures 
beyond a BDO manager review. Of the 15 airports tested, 1,420 of the 18,152 
(8 percent) referral reports recorded in FY 2012 bypassed management review. 
Five of the airports had more than 15 percent of the referral reports bypass 
management review. Only 1 of the 15 airports tested had 100 percent 
management review of referral reports recorded in PMIS. 
BDOs are required to document information regarding each referral, including 
the reason(s) for a referral, the BDOs involved, and the resolution. However, 
passenger-specific data are prohibited from being recorded. 

We assessed more than 110,000 referral records in PMIS from April 1, 2009, 
through September 30, 2012. Of those records— 

• 7,019 did not identify the primary or secondary BDO; 

• 1,194 did not meet the criteria for a referral; 

• 442 were referral records that were deleted; and 

• 143 did not contain a code for the airport where the referral was made. 

Additionally, we identified duplicate records and one record that contained 
personally identifiable information. 

Incomplete and inaccurate PMIS SPOT referral data may have been used to 
present program results to TSA senior leadership and oversight officials to 
illustrate the results of the SPOT program (see Appendix D for FY 2012 PMIS 
SPOT referral data). For example, 4 months after providing our audit team with 
PMIS data, the SPOT program office identified and corrected errors in the 
database before providing similar data to our investigators, who were 
conducting a separate investigation. We were not informed that the original 
data we received had been changed. SPOT program officials said they corrected 
errors in the level of LEO involvement and deleted duplicate records. 
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Training 



TSA has not developed a training strategy that addresses the goals and 
objectives of the SPOT program. For example, BDO formal training, such as 
refresher training, is not consistently provided. Additionally, there is no 
formalized process to evaluate BDO instructors, who provide the only formal 
classroom training to BDOs, in the Performance Accountability and Standards 
System. 6 According to GAO, a well-designed training program should be linked 
to the agency's goals and to the organizational, occupational, and individual skills 
and competencies needed for the agency to effectively perform. 7 As a result, 
TSA cannot ensure that training contributes to the uniform screening of 
passengers. 

Formal Training 

TSA does not consistently offer formal refresher training to BDOs. Beginning in 
May 2006, all BDOs were required to attend the SPOT Basic Training course for 
BDO certification. TSA's training task analysis emphasized the importance of 
recurring training when it reported: 

...observation skills are among the perishable variety. They need to be 
constantly honed and refocused on some regular basis. Observation is 
the single most important task in the entire SPOT Program. ...but little 
training is provided to address its importance, and there is virtually no 
measurement of the skill in the current testing program. 8 

TSA did not start providing refresher training for currently certified BDOs until 
May 2011. The 3-day refresher training consists of a review of SPOT Standard 
Operating Procedure requirements and addresses deficiencies in prior basic 
training courses. TSA determined that BDOs were not receiving training on 37 of 
the 63 (approximately 59 percent) required job tasks in the original versions of 
the basic training course (see Appendix F for SPOT program training task analysis 
executive summary). 



6 The Performance Accountability and Standards System is a pay-for-performance management system 
that promotes and sustains a culture of high performance and accountability. It is designed to ensure that 
employees know what they need to do to accomplish their work successfully and to help TSA accomplish 
its mission through the use of a pay-for-performance system. 

7 U.S. General Accounting Office, GAO-04-546G, Human Capital: A Guide for Assessing Strategic Training 
and Development Efforts in the Federal Government, March 2004. 

8 Transportation Security Administration, Training Task Analysis for the Screening of Passengers by 
Observation Techniques Program, December 16, 2011. 
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BDOs with a year or more of experience are eligible to complete TSA's refresher 
training. According to TSA, 713 of the approximately 2,200 eligible BDOs 
(approximately 32 percent) have received refresher training in the past 2 years. 
Of the 101 BDOs interviewed, 88 were eligible to receive the refresher training. 
Of the 88 eligible BDOs, 65 (approximately 74 percent) have not received the 
training. The SPOT program office reported that because of training staffing 
constraints, they needed to prioritize the training of new BDOs before 
conducting refresher training. As a result, BDOs who have not received the 
refresher training may be operating at varying levels of proficiency. 

Instructors 

BDO instructors provide the only formal classroom training to BDOs. During July 
2012, TSA provided training to BDO instructors and tested them on their teaching 
abilities. TSA identified six BDO instructors who did not have the instructor 
knowledge, skills, or abilities to instruct BDO classes. TSA provided remedial 
instruction to those instructors in order for them to become qualified to teach. 
However, TSA does not evaluate BDO instructors on their instructional abilities in 
their Performance Accountability and Standards System. Additionally, TSA does 
not have a program to provide recurrent training to BDO instructors. Therefore, 
the program office cannot ensure that BDOs are effectively and consistently 
trained. 

Outreach and Engagement 



The relationship between BDOs and local LEOs needs to be improved. TSA 
incorporated law enforcement response as an integral part of the SPOT program. 
However, the SPOT program office has not ensured that airports effectively 
engage local law enforcement. BDOs and local LEOs at the airports contacted 
said that there was insufficient understanding of the roles and responsibilities 
that each had relative to the SPOT program. For example, LEOs at seven airports 
contacted said that they had not received clear information about BDO duties 
and why referrals from BDOs warranted law enforcement response. Conversely, 
BDOs expressed concerns about the consistency of LEOs' responses to referrals. 
BDOs said that local LEOs did not consistently respond to referrals or engage 
referred passengers. TSA data show that LEOs did not respond to approximately 
2 percent of the referrals between October 2011 and September 2012. 9 
Additionally, TSA data indicate that LEOs did not question approximately 
13 percent of referred passengers during that same period. 



This information is based on PMIS data, which may be incomplete and inaccurate because of errors 
identified. 
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Of the 15 airports we contacted, 3 had locally developed LEO outreach activities. 
At the airports using outreach activities, BDOs and LEOs reported more 
consistent and effective working relationships. The success of the program may 
be affected if BDOs and LEOs do not collaborate effectively. 

Financial Accountability 



The SPOT program's financial plan does not include priorities, goals, objectives, 
or financial performance measures. According to the SPOT program office, the 
program was allocated more than $1 billion between FYs 2007 and 2012. The 
data provided by TSA indicate that the program expended an estimated 
$878 million for the program office and SPOT personnel. OMB Circular A-ll 
describes budget formulation, development, and execution requirements that 
include needs analysis and budget development, budget execution, and 
expenditures tracking. Budget control is an integral part of an entity's planning, 
implementing, reviewing, and accountability for stewardship of government 
resources and achieving effective results. 10 Because the SPOT program did not 
have a financial plan that included priorities, goals, objectives, or measures, TSA 
cannot (1) show that SPOT was cost-effective, (2) identify opportunities for 
improvement, or (3) justify the program's expansion. 

Prior to FY 2012, the SPOT program office did not fully determine priorities for 
future spending or develop an itemized forecast of future funding and 
expenditures. Performance information was not used to assess the effectiveness 
of program activities to develop budget priorities. According to program 
officials, the program's historical funding was the basis for spending estimates. 
Beginning in FY 2012, the Behavior Detection and Analysis Division developed 
spend plans that identified project funding requirements. According to the 
FY 2013 draft spend plan, the objective of the spend process is to collect, 
identify, and document funding requirements for the budget year with a 5-year 
forecast. However, the spend plans did not include a comprehensive accounting 
of SPOT funding requirements, such as BDO costs and training. 

Resources 

BDOs at 12 airports contacted were used for activities not associated with their 
primary behavior detection duties. For example, nine airports have used BDOs 
as periodic travel document checkers to review passenger boarding passes and 
identification at screening checkpoints. The SPOT Standard Operating Procedure 



U.S. General Accounting Office, GAO/AIMD-00-21.3.1, Standards for Internal Control in the Federal 
Government, November 1999. 
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allows airports to divert BDOs to travel document checks during times of peak 
passenger traffic or other urgent circumstances. Additionally, one airport used 
BDOs to augment travel document checkers in order to identify passengers 
selected for additional screening. Airport officials explained that they used BDOs 
because Transportation Security Officer (TSO) travel document checkers were 
not identifying passengers who needed additional screening. SPOT program 
officials did not analyze airports' use of BDOs for activities not associated with 
behavior detection duties to evaluate program performance and financial 
impact. By diverting BDOs from their behavior detection duties, TSA may be 
decreasing the effectiveness of a defined layer of security and are not using BDO 
staff as intended. 

BDO Selection, Allocation, and Performance 



The SPOT program needs to ensure that it has an effective process to identify 
and address challenges that may affect the success of the program. 

TSA senior airport officials at airports contacted raised concerns regarding 
selection, allocation, and performance of BDOs. These issues may directly affect 
the success of the SPOT program. Additionally, TSOs are the only source of 
candidates for BDO positions. Limiting BDO recruitment may not provide the 
program with the highest-quality candidates. 

TSA officials at the airports contacted said that they did not understand how 
BDOs are allocated to airports. Federal Security Directors at the airports 
contacted were unsure whether resources were allocated to airports solely 
based on the number of passengers boarding planes; or if airport layout, queuing 
design, and scheduling constraints were also considered. Without a clear 
understanding of allocation methods and the opportunity to provide input on 
BDO allocation, TSA officials at airports may find resources limited to the point 
where the expected benefits of this security layer may not be realized. 

TSA does not use an evaluation period to determine whether new BDOs can 
effectively perform behavior detection and analysis duties. TSA also does not 
have a process to move BDOs back to their TSO positions without penalty. This 
may result in a workforce with some BDOs not able to perform their duties 
effectively. 

Conclusion 

Strategic planning is the keystone to a successful program. Because OMB 
Circular A-ll guidance identifies requirements for agency strategic planning, it 
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would be prudent for agency programs to follow these same principles to help 
ensure program success and contribute to the agency's mission. Without the 
implementation of a SPOT strategic plan that contains key elements, TSA cannot 
ensure that passengers at U.S. airports are screened in an objective manner, 
show that the program is cost-effective, or reasonably justify the program's 
expansion to ensure that threats to aviation security are effectively prevented. 11 



Recommendations 



We recommend that the Assistant Administrator, Office of Security Capabilities: 



Recommendation #1: 



Develop and implement a comprehensive strategic plan for the SPOT program 
that includes— 



• Mission, goals, objectives, and a system to measure performance; 

• A training strategy that addresses the goals and objectives of the SPOT 
program; 

• A plan to identify external partners integral to program success, such as 
law enforcement agencies, and take steps to ensure that effective 
relationships are established; and 

• A financial plan that includes identification of priorities, goals, objectives, 
and measures; needs analysis; budget formulation and execution; and 
expenditure tracking. 



Recommendation #2: 



Develop and implement controls to ensure completeness, accuracy, 
authorization, and validity of referral data entered into the Performance 
Measurement Information System. 



Recommendation #3: 



Develop and implement a plan that provides recurrent training to BDO 
instructors and BDOs. 



The Government Performance and Results Act of 1993 requires agencies to develop a strategic plan for 
program activities that contains (1) a comprehensive mission statement, (2) general goals and objectives, 
(3) how goals and objectives are to be achieved, (4) how performance plan goals relate to the general 
goals and objectives in the strategic plan, (5) external factors that could significantly impact achievement 
of the general goals and objectives, and (6) a description of the program evaluations used in establishing 
or revising general goals and objectives, with a schedule for future program evaluations. 
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Recommendation #4: 

Develop and implement a plan to assess BDO instructor performance in required 
core competencies on a regular basis. 

Recommendation #5: 

Monitor and track the use of BDOs for non-SPOT related duties to ensure BDOs 
are used in a cost-effective manner and in accordance with the mission of the 
SPOT program. 

Recommendation #6: 

Develop and implement a process for identifying and addressing issues that may 
directly affect the success of the SPOT program such as the selection, allocation, 
and performance of BDOs. 

Management Comments and OIG Analysis 

TSA provided comments to this draft report and a copy of the response in its 
entirety is included in Appendix B. TSA also provided technical comments, which 
we incorporated as appropriate. 

According to its response to the draft report, TSA agreed with our 
recommendations for improvements to the SPOT program. Below are TSA's 
responses to each recommendation and our analysis. 

TSA Response to Recommendation #1 : TSA concurred with the recommendation; 
however, it disagreed with our assessment that it had not implemented a 
strategic plan to ensure the SPOT program's success, or that it had not 
developed performance measures for the program. 

According to TSA's response, it is in the process of updating its strategic plan to 
reflect the organization's recent realignment. The strategic plan will elaborate 
on the Behavior Detection and Analysis Program's efforts to work with external 
partners such as law enforcement. Additionally, TSA will execute a BDO training 
strategy elaborating on many of the efforts that are currently under way. Finally, 
the Behavior Detection and Analysis Program will continue to produce a yearly 
spend-plan that identifies goals, objectives, budget formulation and execution, 
and expenditure tracking. 
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OIG Analysis : Our report recognizes that TSA had a draft strategic plan and 
performance metrics plan. However, these documents were not approved or 
fully implemented at the time of our review. 

TSA's planned and current actions are consistent with the intent of the 
recommendation. This recommendation is resolved, but will remain open until 
TSA provides its updated strategic plan for our review and has demonstrated 
progress in implementing the plan. 

TSA Response to Recommendation #2 : TSA concurred with the recommendation 
and said that in December 2012, it completed an audit of all SPOT referrals data 
entered into PMIS between March 2010 and August 2012. Additionally, 
mandatory guidance was provided to airports in order to correct technical and 
administrative errors. TSA officials stated that the entire audit and remediation 
process will be completed by July 2013. 

TSA has two information technology initiatives under way to enhance the 
accuracy and validity of SPOT data. The first initiative, which is scheduled to be 
implemented in May 2013, will include turning off the ability for SPOT referral 
reports to automatically bypass manager review after 72 hours. The second 
initiative creates a new SPOT database that will be more flexible, reduce data 
entry requirements, and have extensive analytical capabilities. The design and 
development phases for the new SPOT database will be completed over the next 
12 months. 

OIG Analysis : TSA's planned and current actions are consistent with the intent 
of the recommendation. This recommendation is resolved, but will remain open 
until OIG— 

• Analyzes the results of the PMIS data audit; 

• Reviews the guidance provided to the airport in order to correct technical 
and administrative errors; and 

• Verifies progress with implementing the information technology initiatives. 

TSA Response to Recommendation #3 : TSA concurred with the recommendation. 
According to TSA's response, it has developed and implemented a plan to 
provide recurrent training for BDO instructors and refresher training for the BDO 
workforce. 

OIG Analysis : TSA's planned and current actions are consistent with the intent 
of the recommendation. This recommendation is resolved, but will remain open 
until TSA provides the training plan for the BDO and BDO instructors, and 
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evidence of its implementation. 

TSA Response to Recommendation #4 : TSA concurred with the recommendation. 
According to TSA's response, it is going to update its National Training Team 
Guide for Instructors and Federal Security Directors in May 2013. The update 
will include a plan to regularly assess BDO instructor performance in core 
competencies. 

OIG Analysis : TSA's planned and current actions are consistent with the intent 
of the recommendation. This recommendation is resolved, but will remain open 
until TSA provides the updated National Training Team Guide for Instructors and 
Federal Security Directors and evidence of its implementation. 

TSA Response to Recommendation #5 : TSA concurred with the recommendation. 
According to TSA's response, it is working on two major initiatives to monitor 
and track the use of BDOs to ensure that they are used in a cost-effective 
manner and in accordance with the mission of the SPOT program. TSA will begin 
a nationwide rollout of a BDO Efficiency and Accountability Metrics Database in 
May 2013. 

OIG Analysis : TSA's planned and current actions are consistent with the intent 
of the recommendation. This recommendation is resolved, but will remain open 
until TSA provides evidence that the BDO Efficiency and Accountability Metrics 
Database and Program Compliance Assessment initiatives have been 
implemented. 

TSA Response to Recommendation #6 : TSA concurred with the recommendation. 
According to TSA's response, it has developed and implemented a number of 
processes for BDOs and BDO managers to provide feedback, suggestions, and 
concerns to the Behavior Detection and Analysis Program office. Within the next 
90 days, TSA plans to capture its communications processes in a BDO 
Communication Plan, which will include an annual survey. 

OIG Analysis : TSA's planned and current actions are consistent with the intent 
of the recommendation. This recommendation is resolved, but will remain open 
until TSA provides its BDO Communications Plan and evidence of its 
implementation. 
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Appendix A 

Objectives, Scope, and Methodology 

The Department of Homeland Security (DHS) Office of Inspector General (OIG) was 
established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment 
to the Inspector General Act of 1978. This is one of a series of audit, inspection, and 
special reports prepared as part of our oversight responsibilities to promote economy, 
efficiency, and effectiveness within the Department. 

The audit objective was to determine whether TSA's SPOT program is structured to 
ensure that passengers at U.S. airports are screened in an objective and cost-effective 
manner to identify potential terrorists. Our original audit objective was to determine 
the extent to which TSA's SPOT program was screening passengers at U.S. airports in an 
objective and cost-effective manner to identify potential terrorism and/or criminal 
activity. However, we could not determine the extent to which the SPOT program is 
operating in an objective and cost-effective manner because the program does not have 
the necessary structure in place to base our assessment. 

To accomplish this objective, we reviewed relevant laws, regulations, and appropriations. 
We also reviewed program policies, procedures, and other documentation, including 
versions of a draft SPOT strategic plan, Standard Operating Procedure guidance, and 
basic training curriculum. We attended the basic SPOT training course given to BDOs to 
gain an overall understanding of the fundamentals of behavior analysis and how it could 
be applied. Additionally, we reviewed other agency's audit reports relevant to the SPOT 
program. 

According to OMB Circular No. A- 11, Preparation, Submission, and Execution of the 
Budget, July 2010, an agency's strategic plan is a valuable tool for communicating a 
vision for the future to managers, employees, delivery partners, suppliers, Congress, and 
the public. We analyzed whether the SPOT program office incorporated the 
characteristics of an effective strategic plan, including (1) a comprehensive mission 
statement, (2) general goals and objectives, (3) how goals and objectives are to be 
achieved, (4) how performance plan goals relate to the general goals and objectives in 
the strategic plan, (5) external factors that could significantly impact achievement of the 
general goals and objectives, and (6) a description of the program evaluations used in 
establishing or revising general goals and objectives, with a schedule for future program 
evaluations. 



We judgmentally selected 15 airports that use the SPOT program (see Table 2). We 
chose these airports by taking into account the following criteria: 



• Number of referrals at each airport for the previous calendar year 
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• Airport category 

• Allocation of BDOs 

• Ratios of BDOs, number of passengers boarding planes, and SPOT referrals 

• Top 10 high-risk airports from DHS's Current Airports Threat Assessment list 

• Physical location (Northeast, Southwest, Central, Pacific Coast) 

We interviewed more than 300 officials at U.S. airports, TSA Headquarters, and external 
agencies. The airport interviews included Federal Security Directors, Assistant Federal 
Security Directors, BDO program coordinators, BDO managers, BDOs, SPOT instructors, 
coordination center personnel, and LEOs. Headquarters interviews included program 
office officials, training division personnel, privacy counsel, SPOT budget officials, 
database administrators, and allocation and scheduling personnel. In addition, we 
interviewed officials from DHS' Science and Technology (S&T) Directorate, U.S. Customs 
and Border Protection, United States Secret Service, and TSA Federal Air Marshal 
Service. Interviews were also conducted with the Federal Bureau of Investigation and 
Israel's El Al Airlines security. 

We obtained and analyzed the PMIS SPOT referral data for April 2009 through 
September 2012. We conducted analysis using IDEA Data Analysis Software to assess 
the accuracy and completeness of referral information collected. Because of insufficient 
internal controls over PMIS data, we did not rely on the data to form conclusions on 
performance of the SPOT program. 

We obtained and analyzed financial information received from the SPOT program office, 
including the pay, compensation, and benefit data it sourced from the National Finance 
Center. We did not test the expenditure data provided by the program office for 
completeness or accuracy, because financial data for the program from FYs 2007 to 
2012 was not readily available. 

This audit did not include work to determine the extent to which the SPOT program is 
based on valid scientific principles for use as an effective layer of aviation security in the 
United States. Additionally, on November 9, 2012, TSA announced that it ratified a 
collective bargaining agreement with the American Federation of Government 
Employees. The agreement will cover approximately 44,000 TSA employees and took 
effect on December 9, 2012. Since the audit was concluded in October 2012, the effect 
of the collective bargaining agreement on the SPOT program was not assessed. 



www.oig.dhs.gov 17 OIG-13-91 




OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 



Table 2. Airport Sample Selection 



Airport 


Category 


BDO 
Allocation 


Referrals from 
Jan. 2011 to 
April 2012 12 


Los Angeles International Airport, 
Los Angeles, CA* 


X 




3,587 


San Francisco International Airport, 
San Francisco, CA 


X 


m 


398 


Orlando International Airport, Orlando, FL 


X 


■ 


6,981 


T^tinn^ 1 ntprnptinn3 1 Airnnrt Tamnp Fl 

1 CI 1 1 1 |JC1 IIILCI lldLlUlldl rAM |JUI I d 1 1 1 |Ja, r 1_ 


1 




1,375 


Hartsfield-Jackson Atlanta International 

Airnnrt Atlanta fnA 

rAM [J W 1 Lf AALId 1 1 Ld, V_J f-\ 


X 


■ 


601 


Chicago Midway International Airport, 

fhiraon 1 


1 


■ 


440 


Chicago O'Hare International Airport, 

fhiraon 1 1 * 


X 




730 


General Edward Lawrence Logan 

I ntprnatinna 1 Airnnrt Ronton MA* 

II 1 LCI 1 Id UUI 1 d 1 f\ 1 1 YJ \J 1 L ^ UUjLUI if IVI i\ 


X 


■ 


2,885 


ThurpnoH Marshall Raltimorp/ Wa^hinpton 

1 1 IUI p.\J\J\A IVIQI Jl f UG 1 11 1 1 IWI 1— 1 V V CI Jl III If^Lwl 1 

International Airport, Baltimore, MD* 


X 


■ 


769 


Detroit Metropolitan Wayne County 
Airport, Detroit, Ml* 


X 


■ 


1,251 


Kansas City International Airport, 
Kansas City, MO 


1 


■ 


108 


John F. Kennedy International Airport, 
New York, NY* 


X 




5,293 


La Guardia Airport, New York, NY 


X 


■ 


1,618 


William P. Hobby Airport, Houston, TX 


1 


■ 


256 


George Bush Intercontinental/ 
Houston Airport, Houston, TX 


X 


■ 


675 



Source: OIG analysis of TSA data 

*Six of the airports in our sample were on TSA's 2010 list of top 10 high-risk airports. 



We conducted this performance audit between April and October 2012 pursuant to the 
Inspector General Act of 1978, as amended, and according to generally accepted 
government auditing standards. Those standards require that we plan and perform the 
audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based upon our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions based upon our 
audit objectives. 



This information is based on TSA's PMIS, which may be incomplete and inaccurate because of errors 
identified. 
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Appendix B 

Management Comments to the Draft Report 



MAY - 3 2013 



MEMORANDUM FOR: 



FROM: 



SUBJECT: 



U.S. Department of Homeland Security 

601 South I2lh Street 
Arlington, VA 20598 

Transportation 
Security 
W Administration 



INFORMATION 

Anne L. Richards 

Assistant Inspector General 

U.S. Department of Homeland Security (DHS) 

John S. Pistole - D „ 
Administrator (J\^ 
Transportation Security Administration 

Response to Draft Report, Transportation Security 
Administration 's Screening of Passengers by Observation 
Techniques (OIG 12-14 1 -A UD-TSA) 



Purpose 



This memorandum constitutes the Transportation Security Administration's (TSA) response to 
the DHS Office of Inspector General (OIG) draft report, Transportation Security 
Administration 's Screening of Passengers by Observation Techniques, March, 20 1 3 . 

Background 

Congressman Beanie Thompson requested an audit of TSA's Screening of Passengers by 
Observation Techniques (SPOT) program to determine its effectiveness, efficiency, and 
economy as a security screening protocol at airports. The audit objective was to determine 
whether TSA's SPOT program is structured to ensure that passengers at U.S. airports are 
screened in an objective and cost-effective manner to identify potential terrorists. OIG 
conducted its audit from April to October of 2012, and interviewed more than 300 officials at 
U.S. airports, TSA headquarters, and external agencies. 

OIG provided TSA with six recommendations for action, all of which TSA concurs with, 
including implementing a strategic plan. The OIG concluded that without the implementation of 
a strategic plan that contains key elements, including mission, goal, and objectives needed to 
develop a system of performance metrics, "TSA cannot ensure that passengers at U.S. airports 
are screened in an objective manner, show that the program is cost-effective, or reasonably 
justify the program's expansion to ensure that threats to aviation security are effectively 
prevented." 

During the audit, TSA provided the OIG with a draft Behavior Detection and Analysis Strategic 
Plan, which included a mission statement, vision statement, goals, and objectives already under 
development. TSA has since finalized this plan (finalized December 2, 2012). TSA also 
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provided the OIG with a draft Performance Metrics P!an (finalized November 2012) which 
establishes the architecture for validated metrics to measure the SPOT program and Behavior 
Detection Officer (BDO) performance. Finally, TSA provided the OIG with a draft return on 
investment study (finalized December 2012), and Program Spend Plans from fiscal years (FY) 
201 1, 2012, and 2013. Of note, TSA provided OIG with approved final documents on December 
5, 2012. These reports detail project descriptions, justifications and impact if not funded along 
with a breakdown of Program Operations and Management costs. 

These documents collectively provide the strategic and financial roadmap for the Behavior 
Detection and Analysis Program (BDAP). While the OIG's report acknowledges the existence 
and submission of these draft plans, TSA would have liked the OIG to acknowledge that TSA 
had begun preparing these documents prior to its audit. Additionally, TSA believes that it has 
more fully implemented the plans than is described in OIG's report and that no significant 
concerns remain unaddressed. 

Discussion 

Earlier this year, consistent with TSA's intelligence-driven and risk-based strategy, the TSA 
Deputy Administrator approved the realignment of BDAP, excluding the research and 
development function, from the Office of Security Capabilities (OSC) to the Office of Security 
Operations (OSO). Placing the operational component of BDAP in OSO aligns behavior 
detection and analysis with other real-lime threat assessment operations. 

The BDAP now has four functional areas: 1 ) strategic planning and policy development; 2) 
training and workforce engagement; 3) compliance and standardization; and 4) analysis and 
metrics. Working in tandem with OSC and the Office of Training and Workforce Engagement 
(OTWE). BDAP focuses on five strategic areas: 

1. Integrate Behavior Detection into Risk-Based Security Initiatives: Continue to 
integrate the behavior detection capability into risk-based security, and study the impact 
that behavior detection and analysis has on the overall effectiveness and efficiency of the 
transportation security system. 

2. Develop Data Management Protocols and Robust Analytical Capabilities: Collect, 
organize, validate, and make data accessible to enable improved decision-making 
regarding BDO allocation, pilots, procedures, and daily operations. 

3. Implement Quality Assurance Oversight Measures: Implement a multifaceted quality 
assurance program for TSA behavior detection and analysis initiatives to ensure 
consistent and standard practices, including airport operation evaluations, current 
training/testing, and data entry/management. 

4. Research, Develop, and Validate New and Existing Behavior Analysis Capabilities: 
Continue to build the scientific foundation for behavior detection and analysis while 
refining current capabilities. Leverage existing and emerging best practices used by other 
domestic and international behavior detection programs as well as law enforcement, 
defense, and intelligence agencies. 

5. Execute Multifaceted Internal and External Communications Plans; Work with 
Federal, state, local and international law enforcement, and civil rights groups to build 
public support and leverage best practices. Engage with the TSA workforce to explain 
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why behavior detection and analysis is a critical layer to the transportation security 
system. 



This framework is consistent with TSA's larger focus on risk-based security, workforce 
engagement and organizational efficiency, and the OSO strategic plan for 2013-2015. 

TSA had begun, and in some cases has completed, all of the recommendations cited by the OIG 
in this report: 

• Strategic Planning and Performance Measurement: TSA finalized a strategic plan 
and performance measurement plan and began implementing many of the objectives in 
these framework documents. 

• Data Accuracy: TSA implemented controls to ensure completeness, accuracy, 
authorization, and validity of referral data entered into the Performance Management 
Information System. 

• Training: TSA implemented a plan to provide recurrent/refresher training to all BDOs 
and BDO instructors, and developed a plan to assess BDO instructor performance in 
required core competencies on a regular basis. 

• Monitor and Evaluate BDO Activity: TSA developed, and implemented an automated 
tool to help evaluate airports' use of BDO resources. 

• Employee Engagement: TSA implemented processes for identifying and addressing 
issues raised by the workforce that may directly affect the success of the SPOT program. 



TSA believes that passengers at U.S. airports are screened by BDOs in an objective manner; 
SPOT is effective and has been validated and determined to identify substantially more high-risk 
travelers than a random screening protocol;'and the program is executed in an efficient manner. 
TSA recognizes the important issues set forth in this report. TSA appreciates OIG's efforts to 
identify areas needing improvement with the SPOT program and will continue to implement 
OIG's recommendations, if not already fully implemented. 



1 SPOT referral report Validation study finat report: Volumes 1-4. Washington, DC: American Institutes for Research, Costigan, T E., 
Makoocn. Z. E., Taylor. T. S„ Sawyer. K., Myers, T. L., ft Toplitz, M. (201 1 ). 



Conclusion 
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U.S. Department of Homeland Security 
Transportation Security Administration (TSA) 



Response to OIG Draft Report 

Transportation Security Administration 's Screening of Passengers by Observation Techniques - 
Sensitive Security Information, March, 2013. 

Recommendation 1 : Develop and implement a comprehensive strategic plan for the SPOT 
Program that includes 1) mission, goals, objectives, and a system to measure performance; 
2) a training strategy that addresses the goals and objectives of the SPOT program; (3) a 
plan to identify external partners integral to program success, such as law enforcement 
agencies, and take steps to ensure that effective relationships are established; and (4) a 
financial plan that includes identification of priorities, goals, objectives, and measures; 
needs analysis; budget formulation and execution; and expenditure tracking. 

TSA concurs. While TSA agrees with this recommendation, it disagrees with the statements in 
the report that "TSA has not implemented a strategic plan to ensure die program's success," nor 
developed performance measures for the program. Although in draft form until December 2012, 
the Program operated under the strategic plan and began implementing many of the strategic 
objectives, including developing a performance metrics plan and continuing to analyze the 
effectiveness of behavior detection and analysis. 

Consistent with Office of Management and Budget (OMB) C ircular A- 1 1 , the SPOT Strategic 
Plan defines its mission, goals, objectives, and actions that the Program will take to realize goals 
and monitor progress, In fact, in November 2012, the Program published and immediately began 
implementing the Behavior Detection and Analysis Division Performance Metrics Plan outlining 
the Program's approach to developing a comprehensive system for effectively measuring SPOT 
performance. Specifically, the performance metrics plan identifies current gaps in performance 
metrics collection, initiatives to address metrics gaps, and resource requirements for the next 3 
years. The plan includes a set of performance metrics that allow for continuous performance 
tracking and evaluation, including: 

• Performance measures for selecting, hiring, and training BDOs; 

• Performance measures for evaluating individual BDO performance and effectiveness; 
and 

• Performance metrics for evaluating program effectiveness as a whole, including which 
behaviors lead to a high probability of a BDO encountering and detecting high-risk 
passengers. 

The Program is in the process of updating the strategic plan to reflect the organization' s recent 
realignment. The Plan will elaborate on the Program's efforts to work with external partners 
such as law enforcement, which the TSA already does on a consistent basis. Additionally, TSA 
will execute a BDO training strategy elaborating on many of the efforts that are currently 
underway. Finally, the Program will continue to produce a yearly spend-plan that identifies 
goals, objectives, budget formulation and execution, and expenditure tracking. In sum, all of the 
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components of a comprehensive strategic plan are in place with efforts underway to achieve full 
implementation. 

TSA finalized a strategic plan and performance measurement plan and began implementing 
many of the objectives in these framework documents. Therefore, TSA requests that OIG 
consider this recommendation implemented and closed. 

Recommendation 2: Develop and implement controls to ensure completeness, accuracy, 
authorization, and validity of referral data entered into the Performance Measurement 
Information System, 

TSA concurs. The Performance Measurement Information System (PMIS) records SPOT 
operational data. The system captures information including arrests and the surrender of 
prohibited items. The Program has taken measures to ensure the completeness, accuracy, and 
validity of SPOT data already submitted into PMIS and developed further controls within the 
system to prevent future errors. 

In December 2012, the Program completed a data audit of all SPOT referrals entered into PMIS 
between March 2010 and August 2012. The Program provided mandatory guidance that airports 
correct technical and administrative errors by amending the PMIS records through a comparison 
with the hard copy referral sheets that are maintained locally. Corrections are documented in 
spreadsheets noting that the PMIS record was changed, when appropriate, or that there was a 
reason why the report was filled out with those fields and thus did not constitute an error. The 
entire audit and remediation process will be finalized by May 2013, for all but five airports, the 
last of which will be completed by July 20 1 3, The Program will continue large-scale data audits 
to ensure the validity and accuracy of SPOT referral data. Additionally, the Program created a 
report for SPOT Transportation Security Managers (STSM) giving them the ability to locally 
monitor and correct errors on a regular basis. 

The Program also has two significant Information Technology initiatives underway to enhance 
the accuracy and validity of SPOT data. The first initiative, to be implemented in the beginning 
of May 2013, modifies the current SPOT module, including turning off the 72-hour auto- 
promote junction and adding drop down menus and check boxes to capture SPOT referral data. 
It also includes additional required fields and business rules to ensure more accurate and 
complete reporting. This initiative will facilitate data integrity in the field and ensure that all 
entries into PMIS have been reviewed and approved by an STSM for quality assurance. BDOs 
and STSMs were trained on this new form the week of April 22-26, 201 3, via Adobe Connect 
sessions, conferences calls, and guidance documents provided on the STSM iShare site, 

The second initiative creates a new SPOT database that will be more flexible, reduce data entry 
requirements to the fullest extent possible, and have extensive analytical capabilities, including 
better trend analysis of indicators resulting in detection of a high risk passenger (system 
requirement phase completed March 2013; design and development phases will be completed 
over the next 12 months). 
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These system modifications provide the necessary controls to ensure completeness, accuracy, 
proper authorization, and validity of the referral data entered into the PMIS system. Therefore, 
TSA requests that OIG consider this recommendation implemented and closed. 

Ri-i'iiinmendation 3 : Develop and implement a plan that provides recurrent training to 
BDO instructors and BDOs, 

TSA concurs. TSA has developed and implemented apian to provide recurrent training for 
BDO instructors and refresher training for the BDO workforce. At the same time, TSA has 
added additional Online Learning Center (OLC) requirements specifically for BDOs. 

BDO Instructors: All National Training Team (NTT) instructors, including those currently 
assigned to the NTT, will attend an academy where they will be taught and evaluated on teaching 
techniques. Specifically, they will receive the Office of Law Enforcement/Federal Air Marshal 
Service Instructor Development Course. They will also receive training on the technical content 
of courses that the NTT teaches including BDO SPOT Basic Training and BDO Refresher 
Training. They are further required to complete Train-the-Trainer (T3) training for the courses 
they will be teaching. Graduates of the NTT Academy deploy as members of the NTT for a 6- 
month period during which they shadow qualified instructors and undergo a mentoring process. 
At any point in the process, instructors failing to meet standards are removed from the qualified 
list of those who can instruct BDO courses. The Office or Training and Workforce Engagement 
(OTWE) and BDAP will collaborate on providing instructors with recurrent training and/or 
guidance if curriculum content changes. 

BDOs: Refresher Training is integral to an effective training program because work conditions 
and policies change over time. Additional or updated training ensures that the program's 
mission continues to be accomplished effectively. During calendar year (CY) 201 1, 232 BDOs 
attended 12 Refresher Training classes; while in 2012, 481 BDOs attended 24 Refresher Training 
classes. BDAP has realigned NTT resources for calendar year 201 3 lo ensure that all eligible 
BDOs 2 complete Refresher Training by the end of December 2013. Tothatend, from February 
11 through March 9, 20 13, NTT members conducted 12reftesher classes for 388 BDOs from 10 
airports. In April 2013, an additional 177 BDOs will be trained during 8 refresher classes. Thus, 
by the end of April 201 3, 1,278 of the approximately 2,020 eligible BDOs will have received 
refresher training. Planning is ongoing to conduct an additional 30 refresher classes during the 
remainder of CY 201 3 to provide training to all eligible BDOs who have not yet received 
refresher training. TSA recognizes that ongoing training on a frequent basis is critical to 
maintaining and enhancing skills learned during BDO SPOT Basic Training. Each year, the 
Program Office will provide a gap analysis to OTWE identifying performance factors and future 
objectives which OTWE will use to update the prior year's Refresher Training, The Program 
will frequently provide the updated training ensuring that BDOs maintain essential skills. 

Additionally, to assist local airports, in January 2013, BDAP launched a series of training 
scenarios for STSMs. These scenarios help STSMs engage with their employees, assist BDOs in 
reviewing critical procedures, and apply critical thinking as a team. Further, OTWE will add 
five new OLC courses to the BDO learning plan during calendar year 2013, and will update two 

2 BDOs are eligible to receive Refresher Training after 4 months. 
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current OLC courses. These OLC courses focus on critical skills in the areas of report writing, 
property searches, Walk the Line procedures, and Causal Conversation techniques. OTWE has 
also begun updating the SPOT Basic and SPOT Refresher Training courses, incorporating 
Standard Operating Procedure (SOP) updates, enhanced instructor notes and instructor best 
practices with an estimated completion date no later than August 201 3. 

The completed and planned refresher training for BDO instructors and BDOs is comprehensive. 
Therefore, TSA requests that OIG consider this recommendation implemented and closed. 

Recommendation 4 : Develop and implement a plan to assess BDO instructor performance 
in required core competencies on a regular basis. 

TSA concurs. On April 2, 2009, the BDAP published a National Training Team (NTT) Guide 
for Instructors and Federal Security Directors (FSD), This training guide provided day-to-day 
operational guidance for members of the NTT. Since this guide did not address the certification 
and recertification of instructors, the Program will issue a new version in May 2013. It will 
include a plan to regularly assess BDO instructor performance in core competencies. The May 
2013 Handbook will provide the following guidelines: 

• All instructors will be evaluated annually in accordance with OTWE's Quality Assurance 
(QA) and Compliance Program OSO-OTT/QACP-SOP-05/20I2 and must receive a QA 
evaluation with a "Meets Expectations'" or better rating. The QA assessment will be 
conducted during delivery of approved technical training courses in the program area in 
which the instructor is qualified to teach. If a QA evaluation rating below "Meets 
Expectations" is received, remediation and re-evaluation will be required. Instructors 
will be allowed a total of three attempts to achieve the "Meets Expectations" rating level. 
The third failure will result in the de-certification of the instructor and removal from the 
NTT. This is consistent with other TSA instruction programs and evaluation processes. 

• Every quarter, all instructors assigned to the NTT will be provided feedback on their 
instructional skills. For classroom instructors, the feedback will align with the 
Transportation Officer Performance System for BDOs. During a BDO Basic Training 
mission, the designated Team Lead will complete a DHS Employee Performance Plan 
and Appraisal Form (EPPA) for BDOs for each instructor assigned to the mission. The 
Team Lead will provide feedback to the instructors along with a signed copy of the 
EPPA. The Team Lead will then provide a copy of the EPPA to BDAP, which BDAP 
will in turn provide to the instructor's Airport of Record (AOR). 

• For Team Leads, the feedback will align with the Lead's Employee Performance 
Management Program (EPMP). STSMs assigned to the NTT shall provide the BDAP 
with a copy of their current EPMP form. For the quarterly feedback, a representative of 
the BDAP will observe the STSM during an NTT mission and, using appropriate 
competencies from the EPMP, provide feedback to the STSM along with a signed copy 
of the EPMP. The EPMP will also be provided to the STSM's AOR. These processes 
are consistent with how employees are evaluated when they are part of TSA National 
Deployment Force. 
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TSA has developed and implemented a plan to assess BIX) instructor performance. Therefore, 
TSA requests that OIO consider this recommendation implemented and closed. 

Recommendation 5 : Monitor and track the use of non-SPOT related duties to ensure 
BDOs are used in a cost-effective manner and in accordance with the mission of the SPOT 
program. 

TSA concurs. The Program is working on wo major initiatives aimed at monitoring and 
tracking the use of BDOs to ensure that they are used in a cost-effective manner and in 
accordance with the mission of the SPOT program. These initiatives take into account that FSDs 
must retain the discretion to employ BDOs to meet a specific security threat or need. The 
initiatives are as follows: 

BDO Efficiency and Accountability Metrics (BEAM) Database: BEAM provides airport 
management and TSA with a standardized tool to track and analyze BDO daily operational data, 
including BDO locations and time spent performing duties. The system streamlines the current 
methods of collecting this data, which varies by each airport. Mote accurate and complete data 
will aJlow for increased standardization of the program and improved analysis of the use of the 
behavior detection capability at airports nationwide. In October 2012, TSA began evaluating the 
tool at select airports refining the system requirements. In May 2013, the Program begins a 
nationwide rollout of BEAM, which will include further pilots, incorporation of updates/lessons 
learned, and procedures mandating airports' use of the system. 

BEAM data will be closely monitored by BDAP providing information that was previously not 
readily available on use of BDOs. Using data from BEAM, TSA will analyze how to deploy 
BDO assets as part of the larger TSA security system, including, but not limited to, whether 
BDO deployment is maximized for operational effectiveness during peak travel times. Through 
analysis, the Program will gain insight to how BDOs arc being regularly utilized, specifically 
how often they are performing duties outside of the scope of SPOT duties. The Program will 
work with other parts of OSO to conduct a cost-benefit analysis on whether these duties provide 
a meaningful addition to TSA's security system and are best carried out by a BDO. The Program 
will regularly report findings to FSDs, Regional Directors (RD), and Other OSO leadership, as 
appropriate and needed. 

Program Compliance Assessment Visits: In 2012, BDAP created the Program Compliance 
Assessment (PCA) team to assess compliance with the SPOT SOP through observation and 
participation in each airport's operation. In addition to SOP compliance, PCAs identify how 
airports are using BDOs via site visits and then validate these observations through BEAM. 

In light of this comprehensive monitoring and tracking plan through BEAM and PCA visits, 
TSA requests that OIG consider this recommendation implemented and closed. 

Recommendation 6: Develop and implement a process for identifying and addressing 
issues that may directly affect the success of the SPOT program such as the selection, 
allocation and performance of BDOs. 
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TSA concurs. TSA has developed and implemented the following processes for BDOs and 
STSMs to provide feedback, suggestions and concerns to BDAP; 

• TSA IdeaFactory: On April 25, 2007, TSA launched the IdeaFactory, a Web-based tool 
that gives more than 50,000 field and headquarters personnel, including BDOs, a voice to 
communicate improvement ideas. IdeaFactory empowers TSA employees to develop, 
promote, and improve innovative ideas for programs, processes, and technologies without 
being filtered through the entire TSA community. Participants submit ideas, provide 
comments on how to improve new concepts, and endorse ideas, which they believe, 
should be recommended for implementation. 

• BDO Discussion Board Blog: All BDOs have access to the SPOT iShare site, which 
includes a discussion board or Blog where they can provide ideas and thoughts. 

• Bi-Weekly Best Practices Conference Calls: In November 201 2, BDAP began bi- 
weekly conference calls with airports providing them with an opportunity to exchange 
best practices, raise concerns, and reinforce key programmatic procedures and policies. 

• SPOT Mailbox: BDAP operates an email box for all BDOs to ask questions, make 
observations or solicit clarification on policies and procedures. The email box provides 
BDOs and STSM with an avenue to clarify policy, recommend new initiatives, and 
acknowledge BDOs' work. The Program aims to respond to each email within one week 
and also provides a weekly summary of all policy and procedures questions and answers 
to all STSMs. This information is shared with employees through airport Shift Briefs. 

« Best Practices iShare Site: In January 2013, TSA launched a Best Practices iShare site, 
which builds upon the success of the biweekly best practices conference calls. At 
airports nationwide, FSDs, SPOT Coordinators, Managers, and BDOs have put 
innovation to work by creating tools that improve their operations. The goal of this site is 
to turn local good ideas into national best practices. 

■ Bi-Weekly Shift Brief: BDAP provides a Shift Brief to STSMs highlighting BDO work, 
including incidents that demonstrate adherence to proper procedures and effective 
outcomes, including, but not limited to, law enforcement intervention. The Brief also 
discusses new initiatives and provides exercises and talking points for Managers on 
critical topics. 

• Weekly Frequently Asked Questions: Frequently Asked Questions are sent out to 
BDO Managers each week via the distribution list summarizing procedural questions 
posed to BDAP over the previous week. 

• Weekly Dashboard; A weekly dashboard is provided to FSDs and field leadership, 
which captures operational metrics from the previous week and includes updates on 
programmatic initiatives. 
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• Monthly Program Calls: Monthly calls are conducted to provide FSDs, SPOT 
Coordinators, and STSMs with updates and notification of upcoming initiatives and 
changes. 

• FCA Visits; PCA methodology will serve as a process for identifying and addressing 
issues that may directly affect the success of the SPOT program. Under the PCA 
Program, PCA Teams comprised of high-performing, specially-trained BDOs will 
observe and assess SPOT operations at airports on an annual basis, with the goal of 
promoting the consistent application of SPOT policies and procedures. The PCA 
Assessments will produce objective, quantitative data that will facilitate the analysis, 
identification, and tracking of operational issues and deficiencies that may require further 
exploration and, potentially, corrective action. Following PCA visits, airports will 
receive a structured report that analyzes the results of the PCA Team's observations and 
identifies any required remediation plans. Additionally, these reports will be available to 
RDs, OSO Division Directors, the OSO Assistant Administrator and others, as necessary. 
The PCAs will also serve as an additional mechanism for BDOs and STSMs to provide 
the BDAP feedback and obtain guidance on SPOT policies and procedures. In the future, 
BDAP anticipates that airports will self-assess using the PCA criteria which will not only 
standardize and improve the quality of self-evaluations, but also further enhance BDAP's 
and airports' ability to identify issues that may affect the success of the SPOT program. 

These efforts and others, including adding an annual BDO survey will be captured in a BDO 
Communication Plan to be completed within the next 90 days. As part of TSA's process of 
updating its strategic plan for the SPOT program, TSA will review current selection, allocation, 
and performance criteria to determine if changes are warranted. 
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Appendix C 

SPOT Referral Process 



Establish environmental baseline 



Assess behavior cues and 
appearance factors 



Perform Walk the Line 



LEO referral 

Complete incident report 
Complete referral report 
^Complete TISS database entry/ 



See Note 1 




Source: TSA 



Note 1: BDOs initiate brief verbal exchanges with passengers waiting in line, while observing passengers' 
behaviors. 

Note 2: The SPOT program assigns point values to specific passenger behaviors and appearances. 
Passengers exhibiting behaviors that exceed the SPOT numerical threshold^ J are referred for 

SPOT referral screening. The SPOT referral screening includes a search of the passenger, the passenger's 
personal property, and a review of the passenger's identification and travel documents. 
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Appendix E 

GAO Status of SPOT Recommendations (as of 4/10/2013) 

(Source: http://www.Kao.ROv/products/GAO-10-763, GAO official website) 

Efforts Validate TSA's Passenger Screening Behavior Detection Program Underway, but 
Opportunities Exist to Strengthen Validation and Address Operational Challenges, 
GAO-10-763, May 20, 2010 

Recommendation: To help ensure that SPOT is based on valid scientific principles that 
can be effectively applied in an airport environment, the Secretary of Homeland Security 
should convene an independent panel of experts to review the methodology of the DHS 
Science and Technology (S&T) Directorate study on the SPOT program to determine 
whether the study's methodology is sufficiently comprehensive to validate the SPOT 
program. This assessment should include appropriate input from other federal agencies 
with expertise in behavior detection and relevant subject matter experts. Status: 
Closed - Implemented 

Recommendation: If research determines that the SPOT program has a scientifically 
validated basis for using behavior detection for counterterrorism purposes in the airport 
environment, then the TSA Administrator should conduct a comprehensive risk 
assessment to include threat, vulnerability, and consequence of airports nationwide to 
determine the effective deployment of SPOT if TSA's ongoing Aviation Modal Risk 
Assessment lacks this information. Status: Open 

Recommendation: If research determines that the SPOT program has a scientifically 
validated basis for using behavior detection for counterterrorism purposes in the airport 
environment, then the TSA Administrator should perform a cost-benefit analysis of the 
SPOT program, including a comparison of the SPOT program with other security 
screening programs, such as random screening, or already existing security measures. 
Status: Open 

Recommendation: If research determines that the SPOT program has a scientifically 
validated basis for using behavior detection for counterterrorism purposes in the airport 
environment, then the TSA Administrator should revise and implement the SPOT 
strategic plan by incorporating risk assessment information, identifying cost and 
resources, linking it to other related TSA strategic documents, describing how SPOT is 
integrated and implemented with TSA's other layers of aviation security, and providing 
guidance on how to effectively link the roles, responsibilities, and capabilities of federal, 
state, and local officials providing program support. Status: Open 
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Recommendation: If research determines that the SPOT program has a scientifically 
validated basis for using behavior detection for counterterrorism purposes in the airport 
environment, then the TSA Administrator should study the feasibility of using airport 
checkpoint-surveillance video recordings of individuals transiting checkpoints who were 
later charged with or pleaded guilty to terrorism-related offenses to enhance 
understanding of terrorist behaviors in the airport checkpoint environment. Status: 
Closed - Implemented 

Recommendation: Concurrent with the DHS S&T Directorate study of SPOT, and an 
independent panel assessment of the soundness of the methodology of the S&T study, 
to ensure the program's effective implementation, and to provide additional assurance 
that TSA utilizes available resources to support the goals of deterring, detecting, and 
preventing security threats to the aviation system, the TSA Administrator should provide 
guidance in the SPOT Standard Operating Procedures or other TSA directive to Behavior 
Detection Officers (BDOs), or other TSA personnel, on inputting data into the 
Transportation Information Sharing System and set milestones and a time frame for 
deploying Transportation Information Sharing System access to SPOT airports so that 
TSA and intelligence community entities have information from all SPOT Law 
Enforcement Officer (LEO) referrals readily available to assist in "connecting the dots" 
and identifying potential terror plots. Status: Closed - Implemented 

Recommendation: Concurrent with the DHS S&T Directorate study of SPOT, and an 
independent panel assessment of the soundness of the methodology of the S&T study, 
to ensure the program's effective implementation, and to provide additional assurance 
that TSA utilizes available resources to support the goals of deterring, detecting, and 
preventing security threats to the aviation system, the TSA Administrator should 
implement the steps called for in the TSA Office of Security Operations Business plan to 
develop a standardized process for allowing BDOs or other designated airport officials to 
send information to TSA's Transportation Security Operations Center about passengers 
whose behavior indicates that they may pose a threat to security, and provide guidance 
on how designated TSA officials are to receive information back from the 
Transportation Security Operations Center. Status: Closed - Implemented 

Recommendation: Concurrent with the DHS S&T Directorate study of SPOT, and an 
independent panel assessment of the soundness of the methodology of the S&T study, 
to ensure the program's effective implementation, and to provide additional assurance 
that TSA utilizes available resources to support the goals of deterring, detecting, and 
preventing security threats to the aviation system, the TSA Administrator should direct 
the TSA Transportation Security Operations Center to utilize all of the law enforcement 
and intelligence databases available to it when running passenger names, for 
passengers who have risen to the level of a LEO referral. Status: Closed - Implemented 
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Recommendation: Concurrent with the DHS S&T Directorate study of SPOT, and an 
independent panel assessment of the soundness of the methodology of the S&T study, 
to ensure the program's effective implementation, and to better measure the 
effectiveness of the program and evaluate the performance of BDOs, the TSA 
Administrator should establish a plan that includes objectives, milestones, and time 
frames to develop outcome-oriented performance measures to help refine the current 
methods used by Behavior Detection Officers for identifying individuals who may pose a 
risk to the aviation system. Status: Closed - Implemented 

Recommendation: Concurrent with the DHS S&T Directorate study of SPOT, and an 
independent panel assessment of the soundness of the methodology of the S&T study, 
to ensure the program's effective implementation, and to better measure the 
effectiveness of the program and evaluate the performance of BDOs, the TSA 
Administrator should establish controls to help ensure completeness, accuracy, 
authorization, and validity of data collected during SPOT screening. Status: Closed - 
Implemented 

Recommendation: Concurrent with the DHS S&T Directorate study of SPOT, and an 
independent panel assessment of the soundness of the methodology of the S&T study, 
to ensure the program's effective implementation, and to help ensure that TSA provides 
BDOs with the knowledge and skills needed to perform their duties, the TSA 
Administrator should establish time frames and milestones for its plan to systematically 
conduct evaluations of the SPOT training program on a periodic basis. Status: Closed - 
Implemented 
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Appendix F 

Training Task Analysis for SPOT Program - Executive Summary 

SENSITIVE SECURITY INFORMATION 




Transportation 
/£ Security 
w Administration 



Training Task Analysis (TTA) for the 
Screening of Passengers by Observation 
Technique (SPOT) Program 

December 16,2011 



Version 3.0 




WARNING: This record conlnins Sensitive Security Information that is controlled under 49 CFR parts 15 mill 1520. No fail of this record may 

be disclosed to persons without a "need to know 17 , as defined in 19 CFR part? 15 and 1 520. except m\'-\ ilic ut Men permission of the 
Administralor of the Transportation Securily Administration, or the Secretary of Transportation. Unauthorized release may result in civil penalty 
or other action. For U.S. Government agencies, pnblie disclosure is governed by 5 U.S.C. 552 and 49 IT'Ii purls 1 5 and 1 520 
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SENSITIVE SECURITY INFORMATION 



Training Task Analysis for the Screening of Passengers by Observation Techniques (SPOT) 



1. Executive Summary 

Following the results of the previously submitted Job Task Analysis (JTA) and Difficulty, Importance, 
and Frequency (DIF) Analysis, the current sequel to the Transportation Security Administration's (TSA) 
Screening of Passengers by Observation Technique (SPOT) Program's effort at improving and updating 
the current training of Behavior Detection Offic ers (BDO) has included expert insights of national trainers 
and Subject Matter Experts (SME) during two workshops that took place in February and March, 2011. 
The SMEs were involved in the functional and hierarchical analyses of (he JTA tasks as part of an overall 
task validation effort. 

The results of the validation process and hierarchical and functional analyses a) identified the trainable 
tasks from the original 110 JTA tasks, bringing the total to 63 trainable tasks; b) grouped the trainable 
tasks into five categories; and c) identified a salient group of communication tasks with three 
subcategories of communication types, depending on the purpose and target of communication. 

With the results of the various analyses that collected direct input from national SPOT SMEs and TS A 
national trainers, the need for a major overhaul of the entire SPOT Program training is evident. It is 
necessary to ensure that the training is current and relevant to emerging threats, both in terms of its 
content as well as in terms of context; that is, the focus of the training delivery system should be shifted to 
exchange, interaction, and facilitation modes rather than the lecture-based foundational information. 

Since May 2011, TSA has rolled out an interim training program that addresses some of the deficiencies 
of its prior training program. As a result, all tasks currently have some degree of coverage in the training 
materials, which is a significant improvement over the previous version. From the 63 tasks, 37 (59%) of 
the tasks were not covered in training materi als prior to the roll-out of the Interim Basi c Training. 

The following calculation of the combinations of behavior, stress, and deception factors illustrates the 
complexity of the BDO's observation tasks. The officers are trained to look for behavioral clusters. By the 
time BDOs make a decision to call a Law Enforcement Officer (LEO), they may encounter any of 63,321 
possible combinations of factors, yielding the threshold value of | m Even prior to making any 
decision, a BD O may en coun ter 151 com binations of factors that^ield| | and 941 combinations of 
factors yielding^ ^| ^ le |H ^1 which calls for SPOT Referral bul does not involve an I.EO 
notification, can be any of 19,079 combinations of factors. 

Newly identified tasks and groups of tasks have emerged as a result of the Job Task Analysis and 
subsequent reviews during the current Training Task Analysis (TTA) effort. In addition, a new subset of 
tasks has been identified that falls under the umbrella of communication tasks and comprises 33 tasks 
(52%). That subset is not addressed in the current training as a separate group of communication tasks. 
Within just the past year, the SPOT program has undergone dynamic changes. In spile of this, the 
program needs to be further refined and aligned with the findings, as discussed in Section 6. 
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To obtain additional copies of this document, please call us at (202) 254-4100, fax your 
request to (202) 254-4305, or e-mail your request to our Office of Inspector General 
(OIG) Office of Public Affairs at: DHS-OIG.OfficePublicAffairs@oig.dhs.gov. 

For additional information, visit our website at: www.oig.dhs.gov, or follow us on Twitter 
at: @dhsoig. 

OIG HOTLINE 

To expedite the reporting of alleged fraud, waste, abuse or mismanagement, or any 
other kinds of criminal or noncriminal misconduct relative to Department of Homeland 
Security (DHS) programs and operations, please visit our website at www.oig.dhs.gov 
and click on the red tab titled "Hotline" to report. You will be directed to complete and 
submit an automated DHS OIG Investigative Referral Submission Form. Submission 
through our website ensures that your complaint will be promptly received and 
reviewed by DHS OIG. 

Should you be unable to access our website, you may submit your complaint in writing 
to: DHS Office of Inspector General, Attention: Office of Investigations Hotline, 245 
Murray Drive, SW, Building 410/Mail Stop 2600, Washington, DC, 20528; or you may 
call 1 (800) 323-8603; or fax it directly to us at (202) 254-4297. 
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